The present invention relates generally to system monitoring, and more particularly, to efficient unified tracing of kernel and user events with multi-mode-stacking
Modern software systems are complex. They are composed of many layers such as application program binaries, application-dependent libraries, third-party libraries, low-level system libraries, and kernels. All of such layers are subject to program bugs and logic errors that can cause performance anomaly in application software. There is a need for a monitoring tool that can monitor the overall system behavior across kernel and user layers. More specific challenges in the system monitoring are as follows.                General purpose mechanism to collect unified information of kernel and user code information        No source code requirement for any code in the software stack        Efficient monitoring        Flexible handling of practical issues such as multiple processes/threads, dynamic fork and kill of processes/threads, and process transformation        
Many debugging techniques have been developed which can be used to solve the above challenges. Conventional user level debugging techniques such as gdb, valgrind, ptrace, and pin [gdb, valgrind, pin] are effective to inspect any code execution by enforcing the program execution in a debugging mode. However, many of these tools impose non-trivial performance overhead due to the debugging mode and it becomes the constraint in the production systems. Also since they are user level tools, they can only observe certain types of kernel level events such as system calls.
Kernel level event tracing has been used to analyze system behavior by using lowlevel operating system kernel events. This technique can investigate performance problems in a more efficient way than the user level tools. However, its downside is that the linkage between kernel evens and high level user code is missing unless the debugging mode is utilized. Thus developers need significant efforts and domain knowledge to understanding the low-level result and apply it for debugging software.
Recently Microsoft™ introduced a closely related technique to this invention that has been developed in parallel. Its technique applies a stack walking mechanism to kernel/user level monitoring for their performance analysis tool called Windows Performance Analyzer (WPA). The major difference of this and the present invention is the monitoring focus. WPA performs stack walking on all processes and all code ranges without any efforts for performance improvement. This difference between the inventive multi-mode stack walking and stack walking of WPA by Microsoft™ is illustrated in FIG. 11.
While WPA has a wide view on the system, such scheme will incur significant overhead in speed and storage. This invention's core contribution is the mechanism to narrow down the monitoring focus to specific application software and to a further finer granularity by using tracing modes.
Accordingly there is a need for improved system monitoring that can monitor overall system behavior across kernel and user layers.